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Requirements for Advanced Multipath in MPLS Networks 


Abstract 


This document provides a set of requirements for Advanced Multipath 
in MPLS networks. 


Advanced Multipath is a formalization of multipath techniques 
currently in use in IP and MPLS networks and a set of extensions to 
existing multipath techniques. 


Status of This Memo 


This document is not an Internet Standards Track specification; it is 
published for informational purposes. 


This document is a product of the Internet Engineering Task Force 


(IETF). It represents the consensus of the IETF community. It has 
received public review and has been approved for publication by the 
Internet Engineering Steering Group (IESG). Not all documents 


approved by the IESG are a candidate for any level of Internet 
Standard; see Section 2 of RFC 5741. 


Information about the current status of this document, any errata, 
and how to provide feedback on it may be obtained at 
http://www.rfc-editor.org/info/rfc7226. 
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Introduction 


There is often a need to provide large aggregates of bandwidth that 
are best provided using parallel links between routers or carrying 
traffic over multiple MPLS Label Switched Paths (LSPs). In core 
networks, there is often no alternative since the aggregate 
capacities of core networks today far exceed the capacity of a single 
physical link or a single packet-processing element. 


The presence of parallel links, with each link potentially comprised 
of multiple layers, has resulted in additional requirements. Certain 
services may benefit from being restricted to a subset of the 
component links or a specific component link, where component link 
characteristics, such as latency, differ. Certain services require 
that an LSP be treated as atomic and avoid reordering. Other 
services will continue to require only that reordering not occur 
within a flow as is current practice. 


Numerous forms of multipath exist today, including MPLS Link Bundling 
[RFC4201], Ethernet Link Aggregation [IEEE-802.1AX], and various 
forms of Equal Cost Multipath (ECMP) such as for OSPF ECMP, IS-IS 
ECMP, and BGP ECMP. Refer to the appendices in [USE-CASES] for a 
description of existing techniques and a set of references. 


The purpose of this document is to clearly enumerate a set of 
requirements related to the protocols and mechanisms that provide 
MPLS-based Advanced Multipath. The intent is to first provide a set 
of functional requirements, in Section 3, that are as independent as 
possible of protocol specifications. A set of general protocol 
requirements are defined in Section 4. A set of network management 
requirements are defined in Section 5. 


1. Requirements Language 


The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", “SHALL NOT", 
"SHOULD", “SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 
document are to be interpreted as described in [RFC2119]. 


Any statement that requires the solution to support some new 
functionality through use of [RFC2119] keywords should be interpreted 
as follows. The implementation either MUST or SHOULD support the new 
functionality, depending on the use of either MUST or SHOULD in the 
requirements statement. The implementation SHOULD, in most or all 
cases, allow any new functionality to be individually enabled or 
disabled through configuration. A service provider or other 
deployment MAY enable or disable any feature in their network, 
subject to implementation limitations on sets of features that can be 
disabled. 
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2. Definitions 


Multipath 
The term "multipath" includes all techniques in which: 


1. Traffic can take more than one path from one node to a 
destination. 
2. Individual packets take one path only. Packets are not 


subdivided and reassembled at the receiving end. 
3. Packets are not resequenced at the receiving end. 
4. The paths may be: 
a. parallel links between two nodes, 
b. specific paths across a network to a destination node, or 


C. links or paths to an intermediate node used to reach a 
common destination. 


The paths need not have equal capacity. The paths may or may not 
have equal cost in a routing protocol. 


Advanced Multipath 
Advanced Multipath is a formalization of multipath techniques 
that meets the requirements defined in this document. A key 
capability of Advanced Multipath is the support of non- 
homogeneous component links. 


Advanced Multipath Group (AMG) 
An AMG is a collection of component links where Advanced 
Multipath techniques are applied. 


Composite Link 
The term "composite link" had been a registered trademark of 
Avici Systems, but it was abandoned in 2007. The term "composite 
link" is now defined by the ITU-T in [ITU-T.G.800]. The ITU-T 
definition includes multipath as defined here, plus inverse 
multiplexing, which is explicitly excluded from the definition of 
multipath. 
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Inverse Multiplexing 

Inverse multiplexing is another method of sending traffic over 
multiple links. Inverse multiplexing either transmits whole 
packets and resequences the packets at the receiving end or 
subdivides packets and reassembles the packets at the receiving 
end. Inverse multiplexing requires that all packets be handled 
by a common egress packet processing element and is, therefore, 
not useful for very high-bandwidth applications. 


Component Link 
The ITU-T definition of composite link in [ITU-T.G.800] and the 
IETF definition of link bundling in [RFC4201] both refer to an 
individual link in the composite link or link bundle as a 
component link. The term "component link" is applicable to all 
forms of multipath. The IEEE uses the term "member" rather than 
"component link" in Ethernet Link Aggregation [IEEE-802.1AX]. 


Client Layer 
A client layer is the layer immediately above a server layer. 


Server Layer 
A server layer is the layer immediately below a client layer. 


Higher Layers 
Relative to a particular layer, a client layer and any layer 
above that is considered a higher layer. Upper layer is 
synonymous with higher layer. 


Lower Layers 
Relative to a particular layer, a server layer and any layer 
below that is considered a lower layer. 


Client LSP 
A client LSP is an LSP that has been set up over one or more 
lower layers. In the context of this discussion, one type of 
client LSP is an LSP that has been set up over an AMG. 


Flow 


A sequence of packets that should be transferred in order on one 
component link of a multipath. 
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Flow Identification 
The label stack and other information that uniquely identifies a 
flow. Other information in flow identification may include an IP 
header, pseudowire (PW) control word, Ethernet Media Access 
Control (MAC) address, etc. Note that a client LSP may contain 
one or more flows, or a client LSP may be equivalent to a flow. 
Flow identification is used to locally select a component link or 
a path through the network toward the destination. 


Load Balance 
Load split, load balance, or load distribution refers to 
subdividing traffic over a set of component links such that load 
is fairly evenly distributed over the set of component links and 
certain packet ordering requirements are met. Some existing 
techniques better achieve these objectives than others. 


Performance Objective 
Numerical values for performance measures: principally 
availability, latency, and delay variation. Performance 
objectives may be related to Service Level Agreements (SLAs) as 
defined in [RFC2475] or may be strictly internal. Performance 
objectives may span links from edge to edge or from end to end. 
Performance objectives may span one provider or multiple 
providers. 


A component link may be a point-to-point physical link (where a 
"physical link" includes one or more link layers, plus a physical 
layer) or a logical link that preserves ordering in the steady state. 
A component link may have transient out-of-order events, but such 
events must not exceed the network’s performance objectives. For 
example, a component link may be comprised of any supportable 
combination of link layers over a physical layer or over logical sub- 
layers -- including those providing physical-layer emulation -- or 
over MPLS server-layer LSP. 


The ingress and egress of a multipath may be midpoint LSRs with 
respect to a given client LSP. A midpoint LSR does not participate 
in the signaling of any clients of the client LSP. Therefore, in 
general, multipath endpoints cannot determine requirements of clients 
of a client LSP through participation in the signaling of the clients 
of the client LSP. 


This document makes no statement on whether Advanced Multipath is 
itself a layer or whether an instance of AMG is itself a layer. This 
is to avoid engaging in long and pointless discussions about what 
constitutes a proper layer. 
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The term "Advanced Multipath" is intended to be used within the 
context described in this document and related documents, for 
example, [USE-CASES] and [FRAMEWORK]. Other Advanced Multipath 
techniques may arise in the future. If the capabilities defined in 
this document become commonplace, they would no longer be considered 
"advanced". Use of the term "advanced multipath" outside this 
document, if referring to the term as defined here, should indicate 
Advanced Multipath as defined by this document, citing the current 
document name. If using another definition of "advanced multipath", 
documents may optionally clarify that they are not using the term 
"advanced multipath" as defined by this document if clarification is 
deemed helpful. 


3. Functional Requirements 


The functional requirements in this section are grouped in 
subsections, starting with the highest priority. 


3.1. Availability, Stability, and Transient Response 


In addition to maintaining stability, limiting the period of 
unavailability in response to failures or transient events is 
extremely important. 


FR#1 The transient period between some service disrupting event and 
the convergence of the routing and/or signaling protocols MUST 
occur within a time frame specified by performance objective 
values. 


FR#2 An AMG MAY be announced in conjunction with detailed parameters 
about its component links, such as bandwidth and latency. The 
AMG SHALL behave as a single IGP adjacency. 


FR#3 The solution SHALL provide a means to summarize some routing 
advertisements regarding the characteristics of an AMG such 
that the updated protocol mechanisms maintain convergence times 
within the time frame needed to meet or not significantly 
exceed existing performance objectives for convergence on the 
same network or convergence on a network with a similar 
topology. 


FR#4 The solution SHALL ensure that restoration operations happen 
within the time frame needed to meet existing performance 
objectives for restoration time on the same network or 
restoration time on a network with a similar topology. 
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FR#5 The solution shall provide a mechanism to select a set of paths 
for an LSP across a network in such a way that flows within the 
LSP are distributed across the set of paths, while meeting all 
of the other requirements stated above. The solution SHOULD 
work in a manner similar to existing multipath techniques, 
except as necessary to accommodate Advanced Multipath 
requirements. 


FR#6 If extensions to existing protocols are specified and/or new 
protocols are defined, then the solution SHOULD provide a means 
for a network operator to migrate an existing deployment ina 
minimally disruptive manner. 


FR#7 Any load-balancing solutions MUST NOT oscillate. Some change 
in path MAY occur. The solution MUST ensure that path 
stability and traffic reordering continue to meet performance 
objectives on the same network or on a network with a similar 
topology. Since oscillation may cause reordering, there MUST 
be means to control the frequency of changing the component 
link over which a flow is placed. 


FR#8 Management and diagnostic protocols MUST be able to operate 
over AMGs. 


Existing scaling techniques used in MPLS networks apply to MPLS 
networks that support Advanced Multipath. Scalability and stability 
are covered in more detail in [FRAMEWORK]. 


3.2. Component Links Provided by Lower-Layer Networks 


A component link may be supported by a lower-layer network. For 
example, the lower layer may be a circuit-switched network or another 
MPLS network (e.g., MPLS Transport Profile (MPLS-TP)). The lower- 
layer network may change the latency (and/or other performance 
parameters) seen by the client layer. Currently, there is no 
protocol for the lower-layer network to inform the higher-layer 
network of a change in a performance parameter. Communication of the 
latency performance parameter is a very important requirement. 
Communication of other performance parameters (e.g., delay variation) 
is desirable. 


FR#9 The solution SHALL specify a protocol means to allow a server- 
layer network to communicate latency to the client-—layer 
network. 


Villamizar, et al. Informational [Page 8] 


RFC 7226 Advanced Multipath Requirements May 2014 


FR#10 The precision of latency reporting SHOULD be configurable. A 
reasonable default SHOULD be provided. Implementations SHOULD 
support precision of at least 10% of the one-way latencies for 
latency of 1 msec or more. 


The intent is to measure the predominant latency in uncongested 
service-provider networks, where geographic delay dominates and is on 


the order of milliseconds or more. The argument for including 
queuing delay is that it reflects the delay experienced by 
applications. The argument against including queuing delay is that 


if used in routing decisions, it can result in routing instability. 
This trade-off is discussed in detail in [FRAMEWORK]. 


3.3. Component Links with Different Characteristics 


As one means to provide high availability, network operators deploy a 
topology in the MPLS network using lower-layer networks that have a 
certain degree of diversity at the lower layer(s). Many techniques 
have been developed to balance the distribution of flows across 
component links that connect the same pair of nodes or ultimately 
lead to a common destination. 


FR#11 In the requirements that follow in this document, the word 
"indicate" is used where information may be provided by either 
the combination of link state IGP advertisement and MPLS LSP 
signaling or via management plane protocols. In later 
documents, providing framework and protocol definitions, both 
signaling and management plane mechanisms, MUST be defined. 


FR#12 The solution SHALL provide a means for the client layer to 
indicate a requirement that a client LSP will traverse a 
component link with the minimum-latency value. This will 
provide a means by which minimum latency performance objectives 
of flows within the client LSP can be supported. 


FR#13 The solution SHALL provide a means for the client layer to 
indicate a requirement that a client LSP will traverse a 
component link with a maximum acceptable latency value as 
specified by protocol. This will provide a means by which 
bounded latency performance objectives of flows within the 
client LSP can be supported. 


FR#14 The solution SHALL provide a means for the client layer to 
indicate a requirement that a client LSP will traverse a 
component link with a maximum acceptable delay variation value 
as specified by protocol. 
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The above set of requirements applies to component links with 
different characteristics, regardless of whether those component 
links are provided by parallel physical links between nodes or by 
sets of paths across a network provided by a server-layer LSP. 


Allowing multipath to contain component links with different 
characteristics can improve the overall load balance and can be 
accomplished while still accommodating the more strict requirements 
of a subset of client LSP. 


3.4. Considerations for Bidirectional Client LSP 


Some client LSPs MAY require a path bound to a specific set of 
component links. This case is most likely to occur ina 
bidirectional client LSP where time synchronization protocols such as 
the Precision Time Protocol (PTP) or the Network Time Protocol (NTP) 
are carried or in any other case where symmetric delay is highly 
desirable. There may be other uses of this capability. 


Other client LSPs may only require that the LSP serve the same set of 
nodes in both directions. This is necessary if protocols are carried 
that make use of the reverse direction of the LSP as a back channel 
in cases such Operations, Administration, and Maintenance (OAM) 
protocols using IPv4 Time to Live (TTL) or IPv4 Hop Limit to monitor 
or diagnose the underlying path. There may be other uses of this 
capability. 


FR#15 The solution SHALL provide a means for the client layer to 
indicate a requirement that a client LSP be bound to a 
particular component link within an AMG. If this option is not 
exercised, then a client LSP that is carried over an AMG may be 
bound to any component link or set of component links matching 
all other signaled requirements, and different directions of a 
bidirectional client LSP can be bound to different component 
links. 


FR#16 The solution MUST support a means for the client layer to 
indicate a requirement that for a specific co-routed 
bidirectional client LSP, both directions of the co-routed 
bidirectional client LSP MUST be bound to the same set of 
nodes. 


FR#17 A client LSP that is bound to a specific component link SHOULD 
NOT exceed the capacity of a single component link. This is 
inherent in the assumption that a network SHOULD NOT operate in 
a congested state if congestion is avoidable. 
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For some large bidirectional client LSPs, it may not be necessary (or 
possible due to the client LSP capacity) to bind the LSP to a common 
set of component links, but it may be necessary or desirable to 
constrain the path taken by the LSP to the same set of nodes in both 
directions. Without an entirely new and highly dynamic protocol, it 
is not feasible to constrain such a bidirectional client LSP from 
taking multiple paths and coordinating load balance on each side in 
order to keep both directions of flows within such an LSP on common 
paths. 


3.5. Multipath Load-Balancing Dynamics 


Multipath load balancing attempts to keep traffic levels on all 
component links below congestion levels if possible and preferably 
well balanced. Load balancing is minimally disruptive (see the 
discussion below this section’s list of requirements). The 
sensitivity to these minimal disruptions of traffic flows within a 
specific client LSP needs to be considered. 


FR#18 The solution SHALL provide a means for the client layer to 
indicate a requirement that a specific client LSP MUST NOT be 
split across multiple component links. 


FR#19 The solution SHALL provide a means local to a node that 
automatically distributes flows across the component links in 
the AMG such that performance objectives are met, as described 
in the prior requirements in Section 3.3. 


FR#20 The solution SHALL measure traffic flows or groups of traffic 
flows and dynamically select the component link on which to 
place this traffic in order to balance the load so that no 
component link in the AMG between a pair of nodes is 
overloaded. 


FR#21 When a traffic flow is moved from one component link to another 
in the same AMG between a set of nodes, it MUST be done so ina 
minimally disruptive manner. 


FR#22 Load balancing MAY be used during sustained low-traffic periods 


to reduce the number of active component links for the purpose 
of power reduction. 


Villamizar, et al. Informational [Page 11] 


RFC 7226 Advanced Multipath Requirements May 2014 


FR#23 The solution SHALL provide a means for the client layer to 
indicate a requirement that a specific client LSP contains 
traffic whose frequency of component link change due to load 
balancing needs to be bounded by a specific value. The 
solution MUST provide a means to bound the frequency of a 
component link change due to load balancing for subsets of 
traffic flow on AMGs. 


FR#24 The solution SHALL provide a means to distribute traffic flows 
from a single client LSP across multiple component links to 
handle at least the case where the traffic carried in a client 
LSP exceeds that of any component link in the AMG. 


FR#25 The solution SHOULD support the use case where an AMG itself is 
a component link for a higher order AMG. For example, an AMG 
comprised of MPLS-TP bidirectional tunnels viewed as logical 
links could then be used as a component link in yet another AMG 
that connects MPLS routers. 


FR#26 If the total demand offered by traffic flows exceeds the 
capacity of the AMG, the solution SHOULD define a means to 
cause some client LSPs to move to an alternate set of paths 
that are not congested. These "preempted LSPs" may not be 
restored if there is no uncongested path in the network. 


A minimally disruptive change implies that as little disruption as is 
practical occurs. Such a change can be achieved with zero packet 
loss. A delay discontinuity may occur, which is considered to be a 
minimally disruptive event for most services if this type of event is 
sufficiently rare. A delay discontinuity is an example of a 
minimally disruptive behavior corresponding to current techniques. 


A delay discontinuity is an isolated event that may greatly exceed 
the normal delay variation (jitter). A delay discontinuity has the 
following effect. When a flow is moved from a current link to a 
target link with lower latency, reordering can occur. When a flow is 
moved from a current link to a target link with a higher latency, a 
time gap can occur. Some flows (e.g., timing distribution and PW 
circuit emulation) are quite sensitive to these effects. A delay 
discontinuity can also cause a jitter buffer underrun or overrun, 
affecting user experience in real-time voice services (causing an 
audible click). These sensitivities may be specified ina 
performance objective. 


As with any load-balancing change, a change initiated for the purpose 
of power reduction may be minimally disruptive. Typically, the 
disruption is limited to a change in delay characteristics and the 
potential for a very brief period with traffic reordering. When 
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configuring a network for power reduction, the network operator 
should weigh the benefit of power reduction against the disadvantage 
of a minimal disruption. 


4. General Requirements for Protocol Solutions 


This section defines requirements for protocol specifications used to 
meet the functional requirements specified in Section 3. 


GR#1 The solution SHOULD extend existing protocols wherever 
possible, developing a new protocol only where doing so adds a 
significant set of capabilities. 


GR#2 A solution SHOULD extend LDP capabilities to meet functional 
requirements. This MUST be accomplished without defining LDP 
Traffic Engineering (TE) methods as decided in [RFC3468]. 


GR#3 Coexistence of LDP- and RSVP-TE-signaled LSPs MUST be supported 
on an AMG. Function requirements SHOULD, where possible, be 
accommodated in a manner that supports LDP-signaled LSP, RSVP- 
signaled LSP, and LSP setup using management plane mechanisms. 


GR#4 When the nodes connected via an AMG are in the same routing 
domain, the solution MAY define extensions to the IGP. 


GR#5 When the nodes are connected via an AMG are in different MPLS 
network topologies, the solution SHALL NOT rely on extensions 
to the IGP. 


GR#6 The solution SHOULD support AMG IGP advertisement that results 
in convergence time better than that of advertising the 
individual component links. The solution SHALL be designed so 
that it represents the range of capabilities of the individual 
component links such that functional requirements are met, and 
it also minimizes the frequency of advertisement updates that 
may cause IGP convergence to occur. 


Examples of advertisement-—update-triggering events to be 
considered include: client LSP establishment/release, changes 
in component-link characteristics (e.g., latency and up/down 
state), and/or bandwidth utilization. 
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When a worst-case failure scenario occurs, the number of 
RSVP-TE client LSPs to be resignaled will cause a period of 
unavailability as perceived by users. The resignaling time of 
the solution MUST support protocol mechanisms meeting existing 
provider performance objectives for the duration of 
unavailability without significantly relaxing those existing 
performance objectives for the same network or for networks 
with similar topology. For example, the processing load due to 
IGP readvertisement MUST NOT increase significantly, and the 
resignaling time of the solution MUST NOT increase 
significantly as compared with current methods. 


5. Management Requirements 


MR#1 


MR#2 


MR#3 


MR#4 


MR#5 


MR#6 


MR#7 


MR#8 


MR#9 


The Management Plane MUST support polling of the status and 
configuration of an AMG and its individual component links and 
support notification of status change. 


The Management Plane MUST be able to activate or deactivate any 
component link in an AMG in order to facilitate operation 
maintenance tasks. The routers at each end of an AMG MUST 
redistribute traffic to move traffic from a deactivated link to 
other component links based on the traffic flow TE criteria. 


The Management Plane MUST be able to configure a client LSP 
over an AMG and be able to select a component link for the 
client LSP. 


The Management Plane MUST be able to trace which component link 
a client LSP is assigned to and monitor individual component 
link and AMG performance. 


The Management Plane MUST be able to verify connectivity over 
each individual component link within an AMG. 


Component link fault notification MUST be sent to the 
management plane. 


AMG fault notification MUST be sent to the management plane and 
MUST be distributed via a link state message in the IGP. 


The Management Plane SHOULD provide the means for an operator 
to initiate an optimization process. 


An operator-initiated optimization MUST be performed ina 
minimally disruptive manner, as described in Section 3.5. 
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7. Security Considerations 


The security considerations for MPLS/GMPLS and for MPLS-TP are 
documented in [RFC5920] and [RFC6941]. This document does not impact 
the security of MPLS, GMPLS, or MPLS-TP. 


The additional information that this document requires does not 
provide significant additional value to an attacker beyond the 
information already typically available from attacking a routing or 
Signaling protocol. If the requirements of this document are met by 
extending an existing routing or signaling protocol, the security 
considerations of the protocol being extended apply. If the 
requirements of this document are met by specifying a new protocol, 
the security considerations of that new protocol should include an 
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evaluation of what level of protection is required by the additional 
information specified in this document, such as data origin 


authentication. 
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